Multisensory context portends object memory

Multisensory processes facilitate perception of currently-presented stimuli and can likewise enhance later object recognition. Memories for objects originally encountered in a multisensory context can be more robust than those for objects encountered in an exclusively visual or auditory context [1], upturning the assumption that memory performance is best when encoding and recognition contexts remain constant [2]. Here, we used eventrelated potentials (ERPs) to provide the first evidence for direct links between multisensory brain activity at one point in time and subsequent object discrimination abilities. Across two experiments we found that individuals showing a benefit and those impaired during later object discrimination could be predicted by their brain responses to multisensory stimuli upon their initial encounter. These effects were observed despite the multisensory information being meaningless, task-irrelevant, and presented only once. We provide critical insights into the advantages associated with multisensory interactions; they are not limited to the processing of current stimuli, but likewise encompass the ability to determine the benefit of one’s memories for object recognition in later, unisensory contexts. The contexts in which learning and retrieval of memories occur can refer to external or internal states (for example, being in a particular room inebriated) or alternatively to fine-grained stimulus features (such as their colour or position) [2,3]. Traditionally, memory has been scientifically investigated under unisensory conditions, despite most real-world situations being multisensory and despite multisensory information oftentimes benefiting perception and thus being well-poised to facilitate memory [4]. Investigations of memory from a multisensory perspective have yielded discrepant results. Some observed that multisensory contexts are beneficial to memory formation and retrieval [1], while others obtained the opposite pattern [1,5]. Individual factors and/or trial-to-trial variability may therefore determine whether multisensory learning will be beneficial. We tested whether multisensory processes related to perceptual salience at one point in time predict an individual’s later memory performance. Participants completed a continuous recognition task involving the presentation of individual images and requiring the discrimination of initial (‘new’) from repeated (‘old’) items. Unbeknown to them, half of the initial images would be presented together with a meaningless sound, resulting in two encoding contexts. Accuracy differences in recognizing (unisensory) image repetitions showed there were individuals who either improved with or were impaired by prior multisensory versus visual encoding (Figure 1A,B). Notably, there was no evidence for a general difference in behaviour between these sub-groups; performance was indistinguishable on both initial and repeated encounters as well as in same and different contexts, arguing against general attention or arousal differences. Rather, only the relative performance difference between types of repeated encounters distinguished participants (significant interaction; Supplemental Information). These accuracy differences were directly linked to processes specifically during the encoding of multisensory information. Electrical neuroimaging analyses of ERPs [6] focused on differences Improving